Place a third of the crackers (about 25) in the bottom of an ungreased 13 inch x 9 inch x 2 inch baking pan. In a saucepan over medium-high heat, melt the butter. Add milk, graham cracker crumbs and sugars; bring to a boil. Boil, stirring constantly, for 5 minutes. Pour half of the mixture over crackers, carefully spreading to cover. Place half of the remaining crackers (about 25) on top. Spread with remaining sugar mixture. Top with remaining crackers. In a saucepan over low heat, stir the peanut butter and chips until melted and smooth. Spread over crackers. Refrigerate until firm, about 1 hour. Cut into small squares.